Method and apparatus of diagnostic test

ABSTRACT

Method, apparatus and computer program for providing a personalized study plan to a learner through cognitive and behavioral diagnosis of the learner. A learner who uses a data input device such as a smart pen and a stylus pen by using data obtained from the data input device. The method, apparatus and computer program relate to technology for obtaining input data based on information inputted by a user for at least one question with the data input device, creating test behavior data on the user from the obtained input data, analyzing cognition and behavior of the user based on at least one of metadata on the at least one question and the created test behavior data, and providing a personalized study plan to the user through an algorithm using machine learning based on the cognition and behavior analysis.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to Korean Patent Application No. KR10-2018-0054280, filed May 11, 2018 and Korean Patent Application No. KR10-2019-0006407, filed on Jan. 17, 2019. The entire contents of theabove applications are incorporated herein by reference.

BACKGROUND

The present invention relates to a diagnostic test, in particular adiagnostic test method, apparatus and computer program providing apersonalized study plan through cognitive and behavioral analysis of alearner who uses a data input device such as a smart pen and a styluspen by using data obtained from the data input device.

Typically, a learner may solve questions of a diagnostic test and aninstructor may grade the learner's answers to diagnose the learnerthrough the learner's performance. With an increasing use of smartdevices such as smartphones and tablet PCs these days, smart pens areincreasingly used as means of input onto a touch screen of a smartdevice and an environment is being created in which a smart pen can beused for learning. This has enabled a learner to write on a particulartype of a printed material with a smart pen, solve a question on theprinted material, or mark an answer. A smart pen can create digital datafrom learner's notes based on recognition of a pattern including the dotpattern developed by Anoto.

Conventional tests of a learner focus only on whether the learner'sanswer is correct or not, without reviewing the process of the learnersolving the question. Since diagnostic tests where a learner uses asmart pen also focus only on whether the learner's answer is correct ornot, it can be said that the diagnostic tests still consist ofevaluation of performances focused on results.

With the diagnostic tests, it is not possible to evaluate a learner byvarious standards. Accordingly, they can only obtain the learner'sperformance result and just provide exercise questions similar to theone that the learner presented an incorrect answer to. In addition,since diagnostic tests where a learner is diagnosed depending only onwhether the learner's answer is correct or not even in an environmentwhere the learner can learn using a smart pen do not sufficiently use avariety of data obtainable from a smart pen, which is a further advancedlearning tool, an improved diagnostic test using it is in demand.

Meanwhile, with the rise of massively open online courses (MOOCs), useof data analysis for personalizing computer-based education is cominginto wider use. However, the MOOCs are used only for a small portion ofthe student community. In particular, most students at the middle andhigh school levels still perform their tasks in an offline(classroom)-based educational environment that uses writing instrumentsand paper as the main tools to take tests. Since there has been no datacollection mechanism operable in the environment, no benefits frompersonalization by data analysis could be had and evaluating testperformances served as a considerable burden. Accordingly, provision ofpersonalized education for a learner through analysis of data of a smartpen is in need.

SUMMARY

Aspects of the present invention has an object of diagnosing a userthrough the user's behavior pattern using a data input device.

The object of the present invention is to provide a system helpingstudents achieve a better test result through a derivation of cognitiveand behavioral factors that affect students' performances and arecommendation engine that can output personalized score improvementstrategies. The system may be implemented through a data-driven andalgorithmically calculated relationship between cognitive and behavioralfactors associated with test-taking behavior and data specific abouteach student that are collected by a data input device such as a smartpen.

In an advancement from existing diagnostic tests focusing on result,aspects of the present invention is intended to perform a diagnostictest focused on process where it is determined whether a learnerefficiently solves a question, solves a question quickly withouthesitation, or has a difficulty “in regard to a particular concept orprocess” and where the student's behavior pattern is compared with thoseof excelling students to find difference between them.

Another aspect of the present invention is intended to perform analysisof cognition and behavior of a learner to determine “why” the learnerexhibits a particular behavior pattern in regard to a particular conceptor process. Provision of a personalized study plan to the learnerthrough the cognition and behavior analysis is intended.

The present invention has been derived to achieve the objects above andsuggests an invention capable of providing a personalized study plan toa user by a diagnostic test using a data input device.

An embodiment of the present invention sets forth a diagnostic testmethod using a data input device, comprising: obtaining input data basedon information inputted by a user for at least one question with thedata input device; creating test behavior data on the user from theobtained input data; and analyzing cognition and behavior of the userbased on metadata on the at least one question and/or the created testbehavior data.

A variety of input devices such as a smart pen and a stylus pen may beused as the data input device. The description below is focused on caseswhere a smart pen is used, but the present invention is not limitedthereto. The diagnosis and analysis can be made in the same manner incases where tests are taken using fingers in mobile devices (e.g.,tablet PCs) or the like.

In a diagnostic test method using a data input device according to anembodiment of the present invention, the input data may comprisecoordinate values of points forming a plurality of strokes andinformation on the time when the points are inputted, and the testbehavior data may comprise a plurality of behavioral metrics.

In a diagnostic test method using a data input device according to anembodiment of the present invention, the analyzing cognition andbehavior of the user may comprise: obtaining test behavior data for theat least one question from each of a plurality of users including theuser; identifying at least one behavioral metric associated with each ofat least one cognitive and behavioral diagnostic (CBD) factor;calculating z-score of the at least one behavioral metric of the user,the z-score being the value of the difference between the value (X) ofthe at least one behavioral metric of the user and the mean value (μ) ofthe at least one behavioral metric associated with the CBD factor of theplurality of users divided by the standard deviation value (σ) of the atleast one behavioral metric associated with the CBD factor of theplurality of users; normalizing the z-score of the least one behavioralmetric; calculating the weighted average of the normalized z-score basedon a predetermined weight for the at least one behavioral metric; anddetermining a value of the CBD factor of the user from the calculatedweighted average.

In a diagnostic test method using a data input device according to anembodiment of the present invention, the at least one CBD factor maycomprise confidence, grit, reasoning, concept memory, deepunderstanding, calculation ability, ability to understand question,test-taking strategy, etc. and each of the CBD factors may be expressedwith a function based on at least one different behavioral metric and/ormetadata on the question.

Another embodiment of the present invention sets forth a diagnostic testmethod using a data input device comprising: obtaining the input databased on information inputted by a user for at least one question withthe data input device; creating test behavior data on the user from theobtained input data; and providing a personalized study plan to the userbased on metadata on the at least one question and/or the created testbehavior data.

A diagnostic test method using a data input device according to anotherembodiment of the present invention may further comprise analyzingcognition and behavior of the user based on metadata on the at least onequestion and/or the created test behavior data.

In a diagnostic test method using a data input device according toanother embodiment of the present invention, the providing apersonalized study plan to the user may comprise: determining a value ofat least one cognitive and behavioral diagnostic (CBD) factor for the atleast one question for a plurality of users including the user;calculating, for each of the at least one CBD factor, a similarity amongat least two questions comprising the at least one question and asimilarity among the plurality of users; calculating, for each of the atleast one CBD factor, a cognitive gap metric by using the similarityamong the at least two questions and the similarity among the pluralityof users; and recommending a question to the user based on thecalculated cognitive gap metric.

In a diagnostic test method using a data input device according toanother embodiment of the present invention, the calculating asimilarity among at least two questions comprising the at least onequestion and a similarity among the plurality of users may compriseapplying a cosine similarity function.

In a diagnostic test method using a smart pen according to anotherembodiment of the present invention, the recommending a question to theuser may comprise: producing the calculated cognitive gap metric foreach of combinations of the user and the at least one question;identifying a question having the highest cognitive gap metric based onthe calculated cognitive gap metric; and recommending the identifiedquestion to the user.

An embodiment of the present invention sets forth a diagnostic testapparatus using a data input device comprising a memory and a processor,wherein the processor is configured to obtain the input data based oninformation inputted by a user for at least one question with the datainput device, create test behavior data on the user from the obtainedinput data, and analyze cognition and behavior of the user based onmetadata on the at least one question and/or the created test behaviordata.

In a diagnostic test apparatus according to an embodiment of the presentinvention, the processor may be further configured to provide apersonalized study plan to the user based on the cognition and behavioranalysis.

In a diagnostic test apparatus according to an embodiment of the presentinvention, the processor may be further configured to obtain testbehavior data for the at least one question from each of a plurality ofusers including the user, identify at least one behavioral metricassociated with each of at least one cognitive and behavioral diagnostic(CBD) factor, calculate z-score of the at least one behavioral metric ofthe user, the z-score being the value of the difference between thevalue (X) of the at least one behavioral metric of the user and the meanvalue (μ) of the at least one behavioral metric associated with the CBDfactor of the plurality of users divided by the standard deviation value(σ) of the at least one behavioral metric associated with the CBD factorof the plurality of users, normalize the z-score of the least onebehavioral metric, calculate the weighted average of the normalizedz-score based on a predetermined weight for the at least one behavioralmetric, and determine a value of the CBD factor of the user from thecalculated weighted average.

In a diagnostic test apparatus according to an embodiment of the presentinvention, the processor may be further configured to determine a valueof at least one cognitive and behavioral diagnostic (CBD) factor for theat least one question for a plurality of users including the user,calculate, for each of the at least one CBD factor, a similarity amongat least two questions comprising the at least one question and asimilarity among the plurality of users, calculate, for each of the atleast one CBD factor, a cognitive gap metric by using the similarityamong the at least two questions and the similarity among the pluralityof users, and recommend a question to the user based on the calculatedcognitive gap metric.

Another embodiment of the present invention sets forth a computerprogram stored in a medium to perform a diagnostic test method using adata input device, wherein the computer program comprises instructionsto cause a computer or a processor to obtain input data inputted by auser for at least one question with the data input device, create testbehavior data on the user from the obtained input data, analyzecognition and behavior of the user based on metadata on the at least onequestion and/or the created test behavior data, and provide apersonalized study plan to the user based on the cognition and behavioranalysis.

A computer program stored in a medium according to another embodiment ofthe present invention may comprise instructions to perform each step ofthe above-mentioned diagnostic test methods using a data input device.

In a diagnostic test method using a smart pen according to an embodimentof the present invention, the creating test behavior data may comprisecalculating a total time of use of the smart pen by the user, and it maycomprise calculating a time of preparation by the user before inputtinginformation on the at least one question and calculating a total time ofinput of information by the user with the smart pen.

In a diagnostic test apparatus according to another embodiment of thepresent invention, wherein to create test behavior data, the processormay be further configured, for the purpose of calculating a total timeof use of the smart pen by the user, to calculate a time of preparationby the user before inputting information using the smart pen andcalculate a total time of input of information by the user with thesmart pen.

By way of the present invention, it is possible to provide a cognitiveand behavioral analysis result and a personalized study plan to alearner using a data input device through cognitive and behavioralanalysis of the learner.

Additionally, the present invention is capable of providing apersonalized study plan to a learner through machine learning algorithmusing artificial intelligence.

BRIEF DESCRIPTION OF THE DRAWINGS

The attached drawings that are included herein and consist of part ofthis specification illustrate exemplary embodiments of the presentinvention and serve to describe characteristics of the invention,together with the above-mentioned general description and the detaileddescription provided below.

FIG. 1 shows a system for providing a diagnostic test using a smart penaccording to various examples of the present invention.

FIG. 2 is a block diagram showing the constitution of the diagnostictest apparatus of the present invention.

FIG. 3 is a flow chart of a diagnostic test method according to variousexamples of the present invention.

FIG. 4 is a diagram showing input data (raw data) obtained from a smartpen according to an example of the present invention.

FIG. 5 is an example of a calculation for creating test behavior datafrom input data obtained from a smart pen according to an example of thepresent invention.

FIG. 6 shows an entire data structure according to an example of thepresent invention.

FIG. 7 is an exemplary diagram showing a result of a diagnostic testaccording to an example of the present invention.

FIG. 8 is an exemplary diagram showing a result of a diagnostic testaccording to an example of the present invention.

FIG. 9 is a flow chart of a diagnostic test method according to anotherexample of the present invention.

FIG. 10 is an exemplary graph illustrating a method for cognitive andbehavioral diagnosis according to another example of the presentinvention.

FIG. 11 is a flow chart of a calculation method for cognitive andbehavioral diagnosis according to another example of the presentinvention.

FIG. 12 shows an exemplary cognitive component calculation for cognitiveand behavioral diagnosis according to another example of the presentinvention.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to theattached drawings. Wherever possible, reference numbers will be usedthroughout the drawings to refer to the parts respectively correspondingto them or parts similar thereto. References made to specific examplesand embodiments are for illustrative purposes only, and are not intendedto limit the scope of the invention or the claims.

Whenever a constituent is referred to as “comprising” anotherconstituent in the description of the invention or the claims, it is notto be construed as consisting of the constituent only, unless otherwisestated, and should be understood as a constituent that possibly furthercomprises other constituents.

Examples of the present invention are hereinafter described in furtherdetail with reference to the drawings.

FIG. 1 shows a system for conducting a diagnostic test relating to anexample of the present invention. A learner can solve at least onequestion presented for the diagnostic test and derive an answer. Inorder to solve at least one question, the learner also may write on theprinted material (20) such as a diagnostic test sheet by using the smartpen (10) as with a pencil or a pen. The learner using the smart pen (10)may hereinafter be referred to as “the user.” In addition, “student,”“learner” and “user” may be used in an interchangeable manner in thefollowing descriptions.

The printed material (20) needs to be one capable of recognizinginformation inputted by the smart pen (10), for example, the one where apattern such as a dot pattern is formed, rather than a piece of ordinarypaper. To this end, a designated file can be manufactured by adesignated printer.

A question for a diagnostic test presented on the printer material (20)is preferably not a question that can be solved by simple memorization.That is, the question is preferably the one where a strategy is requiredto solve it and the solving process needs to be stated in order to reacha conclusion. For example, a type of question with a narrow choice ofstrategy such as calculation may be useful for analyzing data fordiagnosis of the learner. However, if a question is too easy for thelevel of the learner, the question may not be appropriate because thelearner may derive the answer upon reading it without using the smartpen. In addition, an answer to a subjective question may be easier fordata analysis than that of an objective question (multiple choice). Ofthe subjective questions, a short-answer question may be easier for dataanalysis than an essay question because an essay question has a broadrange of answers depending on how students express them.

The smart pen (10) as a data collecting device may comprise a camera(not shown) for recognition of a pattern inputted in the printedmaterial (20) and a short distance communication module for transmittingdata to the diagnostic test apparatus (30). This may enable real-timetransmission of data from the smart pen (10) by streaming. When adiagnostic test is conducted with the smart pen (10), it is necessary tocheck whether or not it is sufficiently charged (90% or more) before useand to check whether or not data are normally transmitted from the smartpen (10).

When the learner solves at least one question in the printed material(20) using the smart pen (10), the diagnostic test apparatus (30) fordiagnosing the learner may carry out a method for diagnostic testsaccording to the various examples of the present invention that aredescribed below in further detail.

Specifically, the diagnostic test apparatus (30) may comprise the memory(31) and the processor (32). By way of the processor (32), each step ofthe diagnostic test method according to the present invention can beperformed and, in addition, a plurality of pieces of information such asdata for diagnostic tests and diagnostic test result data, for example,may be stored in the memory (31) connected to the processor. Thediagnostic test apparatus (30) may be, for example, a mobile device(e.g., a tablet PC) or a computer paired with the smart pen (10). It mayalso be connected to a server or used as a server. Not all constituentsof the diagnostic test apparatus (30) are shown; it is well known to askilled person in the art that it may comprise a communication module, auser interface, a display, etc.

Meanwhile, the diagnostic test methods described herein may be carriedout by a computer or a processor using a computer program stored in amedium. That is, the computer program stored in a medium according tothe present invention may comprise instructions that cause hardware,such as a computer or a processor, to perform the methods describedherein.

FIG. 2 is a block diagram showing a constitution of the diagnostic testapparatus (30) of the present invention. The diagnostic test apparatus(30) may comprise the test diagnostic module (35), the cognitive andbehavioral diagnostic module (36), and the personalized learning module(37), which perform the diagnostic test methods described herein.

The test diagnostic module (35) may perform each step for conducting thediagnostic test method (300) using a smart pen. That is, the diagnostictest apparatus carrying out the diagnostic test method (300) describedhereinafter may be the test diagnostic module (35).

With reference to FIG. 3, the diagnostic test method (300) using a smartpen is described in further detail as below.

First, a user may enter information such as question solving with thesmart pen on the printed material for at least one question presentedfor the diagnostic test. Then, the diagnostic test apparatus may performa step (S301) of obtaining input data of the smart pen based on theinformation inputted by the smart pen.

The step of obtaining input data of the smart pen may comprise a step ofidentifying a page inputted by the user with the smart pen and a step ofidentifying at least one of a plurality of strokes inputted on the page,coordinate values of points forming each of the plurality of strokes,information on time when the points are inputted, and writing pressureof the points.

For example, FIG. 4 shows input data that may be obtained from the smartpen. A plurality of strokes (Stroke 1, Stroke 2, Stroke 3, and Stroke N)on a particular page may be identified. For example, a stroke may bespecified by a continuous trajectory from a pen down point to a pen uppoint of the smart pen, and the coordinates from the pen down point tothe pen up point may be recorded at specific time intervals (e.g., 0.012to 0.015 in FIG. 4). Accordingly, input data including XY coordinatevalues of points forming a stroke, information on the time when thepoints are inputted, and writing pressure of the points can beidentified for each stroke. In an example shown in FIG. 4, it isunderstood that, in Stroke 1 among the plurality of strokes of theparticular page, the initial point has the (X, Y) coordinates of(051.94, 053.70) and the writing pressure of 070 and was inputted onOct. 26, 2017 at 11:13:52:624. As shown above, the number of all strokesand detailed data on each stroke may be extracted for each page.

Next, the diagnostic test apparatus may perform a step (S302) ofcreating test behavior data on the user from the obtained input data, asanalysis information on the smart pen user associated with the at leastone question.

Thus, the analysis information on the user associated with the at leastone question may be referred to as the test behavior data on the user.The test behavior data (analysis information) may be created for eachquestion in association with the at least one question and may becreated for a plurality of users that can be identified by name, age,gender, etc. The test behavior data may comprise a plurality ofbehavioral metrics. Examples of the plurality of behavioral metricsinclude delay time, stroke length, count of pauses in input of specificdurations, input speed at specific stages of testing, length of input,and rework but are not limited thereto.

As exemplarily shown in FIG. 5, the diagnostic test apparatus considersstroke inputs as points in the two-dimensional Euclidean plane from thecollected input data (e.g., stroke positions and time stamps) andapplies a Cartesian geometric formula to calculate the distance andspeed of input, thereby creating behavioral metrics. An example ofbehavioral metrics using the two stroke points (x1, y1) and (x2, y2)shown in FIG. 5 may be calculated as follows.

The distance between the two stroke points (x1, y1) and (x2, y2) may becalculated using the formula below:

length=√{square root over ((x ₂ −x ₁)²+(y ₂ −y ₁)²)}

The time between the two stroke points is given by:

time duration=t ₂ −t ₁

The input speed is given by:

${{input}{speed}} = \frac{\sqrt{\left( {x_{2} - x_{1}} \right)^{2} + \left( {y_{2} - y_{1}} \right)^{2}}}{t_{2} - t_{1}}$

Alternatively, the test behavior data are described in further detail asbelow.

Delay time may be determined by the difference in the time stampsbetween the last stroke of one character and the first stroke ofsubsequent character. Count of pauses in input of specific durations maybe determined by counting the number of time intervals during which nostroke is inputted. Length of input may be determined by the sum of thelengths of the strokes of all characters inputted by the user. Reworkmay be determined by a movement in the negative x-axis and/or they-axis.

For the purpose of creating test behavior data for the user, thediagnostic test apparatus may calculate the number of all strokes (Totalstroke) extractable from input data and calculate a total time of use ofthe smart pen by the user (Total time). A total time of use of the smartpen by the user for at least one question (Total time) may include atime when the user reads and deliberates on the at least one questionafter it is presented (Intro time) and a time when the question isactually solved with the smart pen (Solving time). Accordingly, thediagnostic test apparatus may calculate the time when the user reads anddeliberates on the question (Intro time) and the time when the questionis actually solved (Solving time). To this end, the time from a pen uppoint of the last stroke of the previous question to a pen down point ofthe first stroke of the next question, for example, may be defined asthe deliberation time (Intro time).

More specifically, in order to create test behavior data on the user,the diagnostic test apparatus may track coordinates of points formingeach stroke and, if the coordinate values of the points staysubstantially the same for a predetermined time, determine that a delayhas occurred. In addition, the diagnostic test apparatus may calculate atotal time of delays and the number of delays (Number of Delays)occurred in association with the at least one question for the user. Thepredetermined time may be, for example, one second, two seconds, threeseconds, four seconds, five seconds, etc. Different weights may beapplied depending on the length of the predetermined time. Accordingly,a total time and the number of delays when the predetermined time is,for example, one second, two seconds, three seconds, four seconds orfive seconds may be calculated.

In addition, the diagnostic test apparatus may additionally determine,as analysis information on the user associated with the at least onequestion, stroke-drawing speed (Stroke velocity, cm/second), initialspeed of stroke (Initiation speed), ending speed (Ending speed), averagespeed (Average speed), solving speed, which is the number of strokes perquestion divided by a solving time per question (Solving velocity,strokes/time), sum of total length of strokes (Ink length), area usedfor solving (Area), complexity in directions of solving progression(Entropy), cross-out (Cross out), number of problems attempted (Problemattempted), number of changes in question solving order (Out of order),time taken to start solving the next question when the order of questionsolving has changed (Out of order time), etc.

For example, area used for solving (Area) may be calculated to be thearea where strokes are present. Complexity in directions of solvingprogression (Entropy) may be calculated by determining that a strokegoing from left to right or from top to bottom is of low entropy and, incontrast, a stroke going from right to left or from bottom to top is ofhigh entropy. Cross-out may be traced by dividing it into cross-out of anumber or a word (typo cross out), cross-out of part of problem solvingprocess (problem solving cross out), and cross-out of the entire solvingor an area corresponding thereto (big cross-out). Problem attempted maybe calculated to be the number of times when a question is attended overa particular time. The number of changes in question solving order (Outof order) may be determined by tracking when the user skips a questionto solve another question.

When the diagnostic test apparatus creates test behavior data includinga plurality of behavioral metrics, the diagnostic test apparatus mayperform a step (S303) of evaluating the user's question solving levelfor the question based on the created test behavior data.

Since the user's question solving level may be evaluated by a behaviorpattern of the user determined by the user's behavior data, the stepabove may be referred to as a step of determining a behavior pattern ofthe user based on the created test behavior data. For example, if it isdetermined that the user has a behavior pattern of “smoothly solvedwithout delay,” the user's question solving level may be evaluated to be“smoothly solved without delay.”

The diagnostic test apparatus may create test behavior data on aplurality of users associated with at least one question and store themin a memory. Alternatively, the diagnostic test apparatus may createanalysis information such as test behavior data on a plurality of usersassociated with the at least one question and transmit it to a separateserver or receive it from the server. The transmission and reception canbe carried out in real time, according to which pieces of analysisinformation such as stored test behavior data may be updatedperiodically. The diagnostic test apparatus may compare pre-stored testbehavior data with test behavior data created for the user inassociation with the at least one question. Instructors desiring toconduct a diagnostic test may share the at least one question on anetwork to accumulate data on a plurality of users, i.e., students,associated with the questions of the diagnostic test.

Additionally, when comparing pre-stored test behavior data with testbehavior data created for the user in association with the at least onequestion, the diagnostic test apparatus may process the created testbehavior data based on correlation among the at least one question. Forexample, when the user has test behavior data that noticeably differfrom the pre-stored test behavior data for specific questions with highdegree of correlation, more attention may be paid to the questionsolving level of that type of the specific questions.

Further, the diagnostic test apparatus may use metadata on testquestions for identification of the areas where the user shows strengthor weakness. Metadata on a question may include information such asdifficulty of the question, subject area of the question, and a properstudent level for the question. Specifically, the test diagnostic module(35) may evaluate the user's question solving level by combining thetest behavior data on the user and the metadata on the question, therebydetermining the user's strengths and weaknesses. An example of theentire data structure that may be used in the test diagnostic module(35) and in the cognitive and behavioral diagnostic module (36) and thepersonalized learning module (37) described below is illustrated in FIG.6. According to FIG. 6, the data structure used in the present inventionmay include not only wring measures (WRITING MEASURES) as the user'stest behavior data and question facts (QUESTION FACTS) as metadata onthe question but also identification information such as basic studentidentification information (STUDENT_ID), question identificationinformation (QUESTION_ID), and test identification information(TEST_ID). It also may include data about test fact information (TESTFACTS) such as test date and place, student fact information (STUDENTFACTS) such as the student's past attendance and performances, andperformance measures (PERFORMANCE MEASURES) representing the percentageof the student's correct responses.

Meanwhile, FIG. 7, which represents a diagnostic test result evaluatingthe user's question solving level based on the user's test behavior datafor a particular question, shows, for example, a specific area of areport card. ‘No.’ indicates a question number; ‘Unit’ indicates asubject area (name of section); ‘Question Point’ indicates pointsassigned to the question; ‘Total Score’ indicates a total score, whichis the sum of ‘OX,’ ‘Concept Score’ and ‘‘Process Score'; ‘OX’ indicatesa score for presenting correct answers; ‘Concept Score’ indicates ascore for application of concepts based on correct understandingthereof; ‘Process Score’ indicates a score for the process of solvingthe question, i.e., a score for how efficiently the question is solvedusing a strategy; and ‘Correct Rate’ indicates the percentage of correctanswers. ‘Understanding’ becomes higher when a behavior pattern of theuser associated with the question evaluated by the test behavior data ismore similar to the patterns of the users who derived the correctanswer. For example, a user who presented the correct answer butdeliberated on the question for a long time or presented an incorrectanswer following an erroneous question solving process and thencorrected the answer after checking has a low score for ‘Understanding.’

As shown in the diagnostic test result (“Diagnostic info Data”) of FIG.7, the diagnostic test apparatus may for example compare, as analysisinformation (test behavior data) on a plurality of users associated withthe first question, the average values of the total number of stokes(Total stroke/N of strokes), the time taken to solve the question(Solving time), the total delay time (Delay time), and the number ofdelays (Number of Delays/N of Delays) of the plurality of users with thetotal number of strokes created (Total stroke), the time taken to solvethe question (Solving time), the total delay time (Delay time), and thenumber of delays (Number of Delays/N of Delays) of a particular userassociated with the first question, respectively.

For example, the average values of the time taken to solve the question,the delay time, the total number of strokes, and the number of delays ofa plurality of users associated with the first question (Q1) arerespectively 44.4, 124.2, 89.3 and 15.3 while the time taken to solvethe question, the delay time, the total number of strokes, and thenumber of delays of the user associated with the first question (Q1) are44.0, 21.2, 88.0, 7.0 respectively. Given this, the fact that the valuesare significantly lower than the average values of the plurality ofusers in the delay time and the number of delays may be considered inevaluation of the user's question solving level.

That is, the diagnostic test apparatus may evaluate a question solvinglevel of the particular user based on the comparison result above. Inthis regard, FIG. 8 shows, as an example, question solving levels of oneor more users based on test behavior data including the total number ofstrokes and the number of delays.

First, in case of a user who is found to have a large number of strokes,a small number of delays, and a short delay time compared withpredetermined criteria, the diagnostic test apparatus may additionallyconsider entropy. Accordingly, it may be evaluate a question solvinglevel of a user having a high entropy to be “worked hard withoutstrategy” and a question solving level of a user having a low entropy tobe “well strategized and recorded solving process carefully.”

Second, in case of a user who is found to have an average number ofstrokes, an average number of delays, an average delay time, thediagnostic test apparatus may consider detailed values of the pieces ofinformation and evaluate the user to be “currently in good understandingbut in need of another test in 2-3 weeks” or “average level and in needof practice.”

Third, the diagnostic test apparatus may evaluate a question solvinglevel of a user found to have a high number of strokes, a high number ofdelays, and a long delay time to be “lacking in sufficientunderstanding.”

Fourth, for a user found to have a small number of strokes, a smallnumber of delays, and a long delay time, the diagnostic test apparatusmay evaluate “solved by mental math” and, for a user found to have asmall number of strokes, a small number of delays, and a short delaytime, the diagnostic test apparatus may evaluate “very familiar with thequestion.”

Lastly, for a user having a small number of strokes and multiple, veryshort delays, it may evaluate “repeatedly checked solving process whensolving question.”

A diagnosis result of a question solving level of a user evaluated asillustrated in FIG. 8 may be displayed as a diagnosis result in the“Behavior Pattern” item shown in FIG. 7. This allows going beyondevaluation of the user simply depending on whether his/her answer to aquestion is correct or not so as to conduct analysis of behavior theuser shows in the question solving process and find causes of thebehavior, which enables a more efficient and effective teaching. Forexample, if a very long delay occurs in a user's question solvingprocess that is longer than a particular time, relevant strokes may bedisplayed in a different color or a delay time may be tagged to the areaof the strokes to more visually express a diagnosis result of the user'sbehavior. With this, an instructor may conceive a more effectiveteaching method to a learner.

As mentioned above, the diagnostic test apparatus may also use metadataon a question. FIGS. 7 and 8 only show a user's question solving levelbased on the user's test behavior data for a particular question.However, when it is combined with metadata on the question, it ispossible to identify the types of questions for which the user hasstrengths or weaknesses. The test diagnostic module (35) may select aspecific type of question (e.g., a question about geometry) particularlyaimed to identify the user's strengths or weaknesses based on metadataon the question.

Meanwhile, the above-mentioned processes regarding FIGS. 3 to 8 may becarried out by the test diagnostic module (35) and descriptions aboutthe test diagnostic module (35) may be applied to the processes.

The cognitive and behavioral diagnostic module (36) and the personalizedlearning module (37) included in the diagnostic test apparatus (30) ofFIG. 2 are described below in further detail.

The cognitive and behavioral diagnostic module (36) and the personalizedlearning module (37) may perform each step for carrying out thediagnostic test method (900) using a smart pen. That is, the diagnostictest apparatus carrying out the diagnostic test method (900) describedbelow may be the cognitive and behavioral diagnostic module (36) and/orthe personalized learning module (37). Alternatively, the cognitive andbehavioral diagnostic module (36) and/or the personalized learningmodule (37) may substantially perform some or all of the steps that thetest diagnostic module (35) performs and may use output of the testdiagnostic module (35).

The diagnostic test method (900) using a smart pen is described as belowin further detail with reference to FIG. 9.

First, for at least one question presented for a diagnostic test, a usermay input information such as question solving on a printed materialwith a smart pen. Then, the diagnostic test apparatus may perform a step(S901) of obtaining input data of the smart pen based on the informationinputted by the smart pen.

Next, the diagnostic test apparatus may perform a step (S902) ofcreating test behavior data on the user from the obtained input data, asanalysis information on the smart pen user associated with the at leastone question. The detailed description relating to steps S301 and S302illustrated in FIG. 3 may also be applied to steps S901 and S902 of FIG.9. In addition, steps S901 and S902 may be performed by the testdiagnostic module (35) and/or the cognitive and behavioral diagnosticmodule (36), and the cognitive and behavioral diagnostic module (36) mayalso receive output data of steps S901 and S902 performed by the testdiagnostic module (35).

Next, the diagnostic test apparatus may perform a step (S903) ofanalyzing cognition and behavior of the user based on the test behaviordata on the user and metadata on the question.

For example, with the cognitive and behavioral diagnostic module (36)such as a cognitive analysis engine, cognitive components such asconfidence, grit, reasoning, concept memory, deep understanding,calculation ability, ability to understand question, test-takingstrategy, focus, creativity, mental math speed, speed of understanding,carefulness, and flexibility may be derived. For example, the componentsmay be defined as below. Grit maybe an indicator showing the degree ofgrit with which a question is solved. Reasoning may be an indicatorshowing whether a test is taken with a logical reasoning. Concept memorymay be an indicator showing whether concepts and formulae are accuratelymemorized and used. Calculation ability may be an indicator measuringcalculation ability, which is one of basic mathematical abilities.Ability to understand question may be an indicator showing whetherinformation in the question is accurately read and interpreted to buildthe right strategy. Test-taking strategy may be an indicator to identifywhether question solving is strategically performed when taking a test.Focus may be an indicator showing an ability to maintain focus throughquestions that need substantial thinking. Creativity may be an indicatorshowing an ability to answer with short/creative responses relative toother students. Speed of understanding may be an indicator showing anability to quickly and correctly understand questions and startanswering. Carefulness may be an indicator of being risk-averse anddouble-checking answers. Flexibility may be an indicator showing anability to successfully course-correct while answering a question.

The cognitive and behavioral diagnostic module (36) may apply dataalgorithms to a combination of the user test behavior data and metadataon a question attempted to show a score card of key underlying cognitivecomponents having a significant impact on the user's performances. Byshowing underlying causes affecting the user's performances, the user isable to implement a more sustainable fix to his/her test behavior.

The cognitive and behavioral diagnostic module (36) may judge “cognitivecomponents” such as confidence, reasoning, and concept memory associatedwith the user's cognition based on the user's test behavior data andmetadata for a question. The cognitive and behavioral diagnostic module(36) may also analyze the user's “behavioral components” for questionsolving based on the user's test behavior data and metadata for aquestion. For example, it can perform judgment on the time when the userreads a question, judgment on the behavior of interpreting the questionor the like based on test behavior data and metadata. That is, thecognitive and behavioral diagnostic module (36) may access “cognitiveand behavioral components” of each student through numerical analysis ofbehavioral metrics calculated from data obtained by using a datacollecting device such as a smart pen, and a cognitive and behavioralcomponent may be referred to as a cognitive and behavioral diagnosticsfactor (CBD factor). Examples of CBD factors and behavioral metricsdependent to them are shown in Table 1.

TABLE 1 CBD Factor Type Behavioral Metrics - Dependencies ConfidenceCognitive f(stroke_gap_time, count_of_long_pauses, writing_speed,initiation_speed, performance) Grit Behavioral f(stroke_length,stroke_time, writing_speed, initiation_speed, performance) ReasoningCognitive f(reasoning_question_tag, total_time, performance) ConceptCognitive f(concept_memory_tag, initial_time, memory performance) DeepCognitive f(concept_application_tag, stroke_length, understandingtotal_time, performance) Calculation Behavioral f(total_time,average_speed, Ability performance) Ability to understand Behavioralf(concept_application_tag, initial_time, question re-work, performance)Test-taking Behavioral f(correct_rate_dip_second_half_vs_second_half,Strategy total_time_dip_second_half_vs_first_half)

Examples of metrics dependent to the CBD factor shown below may beconsidered.

Confidence

stroke_gap_time (total sum of time gaps between strokes when writing isnot in progress): If this value is high, it may be considered that muchdeliberation was made when solving the question.

Count_of_long_pauses (total number of long deliberations): This value ishigh when the number of long deliberations is small and the value is lowwhen the number of the deliberations is small.

Writing_speed (writing speed when solving the question): This considersnot only comparison of speed among students but difference in speed ofsolving each question as the test progresses. Relatively confident andquick question solving results in a high value and a solving that is notresults in a low value.

Initiation_speed (writing speed when the question solving begins): Thisindicates the speed of understanding the question and initiation of thequestion solving. For most students, the initiation speed of solving aquestion that they are familiar with and confident about is fast.

Grit

Stroke_length (total length of writing in the question solving): A highvalue means a relatively large amount of question solving.

Stroke_time (sum of the total time of writing): A longer solving timemeans a higher value.

Writing_speed (writing speed of question solving): Relatively confidentand quick solving results in a high value and a solving that is notresults in a low value.

Initiation_speed (writing speed when question solving begins): For moststudents, the initiation speed of solving a question that they arefamiliar with and confident about is fast.

Test Strategy

FIG. 10 lists the question numbers in the order of solving the questionson the x-axis, and shows the time taken to solve each question on they-axis. By, for example, calculating the percentage of correct answersand the amount of time taken for the first half and second half of thequestions, an indicator to confirm whether the student strategicallysolves the questions that he/she can solve before other may becalculated.

Detailed contents on how to derive the CBD factor from the dependentmetrics are shown below.

Step 1

For a set of responses for each unique question, the cognitive andbehavioral diagnostic module (36) may, at the student level, determinez-score of associated behavioral metrics. This is to determine astudent's relative performance for a particular metric. Z-score may begiven by the following formula:

$z = \frac{X - \mu}{\sigma}$

wherein X is the student's behavioral metric value for the question, μis the mean value of the behavioral metrics across all responses to thequestion, and σ is the standard deviation of the behavioral metricsacross all responses to the question.

Step 2

The cognitive and behavioral diagnostic module (36) may normalize thez-score to a particular scale. For example, the cognitive and behavioraldiagnostic module (36) may normalize the z-score to the scale of 1-10.This is to facilitate comparison of z-scores for different metrics. Anormalized z-score may be given by the following formula:

${{{Normalized}z} - {score}} = {1 + {\left( {z - {\min(z)}} \right)*\frac{10 - 1}{{\max(z)} - {\min(z)}}}}$

wherein min(z) is the minimum value of the z-score for the behavioralmetric among the set of responses, and max(z) is the maximum value ofthe z-score for the behavioral metric among the set of responses.

Step 3

The cognitive and behavioral diagnostic module (36) may calculate thefinal score of a student for a particular CBD factor from a weightedaverage of the z-scores for various components of the CBD factor. Withthis, using the module, the system of the present invention may providean in-depth understanding of cognitive factors that govern performancesof the user. Accordingly, the cognitive and behavioral diagnostic module(36) may aim to cure a root cause of performance gaps, not simplyresolving a superficial symptom.

The steps above are described as below with reference to FIG. 11.

The cognitive and behavioral diagnostic module (36) may perform a step(S1001) of filtering unique questions having a particular tag based onmetadata on the questions. For example, when a plurality of usersanswer, using a smart pen, the unique questions having a particular tag,the obtained answers may be classified into correct answers andincorrect answers. For example, test behavior data may be obtained fromeach of the plurality of users. The step of filtering unique questionshaving a particular tag based on metadata on the questions may beskipped.

The cognitive and behavioral diagnostic module (36) may perform a step(S1002) of calculating z-score for each of at least one metric includedin a predetermined functional formula associated with a CBD factor inorder to derive the CBD factor. For example, z-score of a particularbehavioral metric of a particular user may be calculated to be the valueof the difference between the value (X) of the particular behavioralmetric of the particular user and the mean value (μ) of the particularbehavioral metrics of a plurality of users divided by the standarddeviation value (σ) of the particular behavioral metrics of theplurality of users.

The cognitive and behavioral diagnostic module (36) may perform a step(S1003) of normalizing the z-score calculated for each of at least onemetric. For example, the cognitive and behavioral diagnostic module (36)may normalize the z-score to the scale of 1-10 or to the scale of 10-1.For example, z-score calculated for each of at least one metric includedin a predetermined functional formula associated with a CBD factor maybe normalized.

The cognitive and behavioral diagnostic module (36) may give a weightedaverage to normalized z-scores for at least one metric to determine theweighted average of the normalized z-scores as the value of the CBDfactor (S1004). In addition, although not illustrated in FIG. 11, theweighted average of the scores may be further weighted (e.g.,age-weighted) based on other factors.

As an example performing the steps illustrated in FIG. 11, FIG. 12 showsa calculation flow to induce “deep understanding” among the cognitiveand behavioral factors of Table 1. Table 1 includes metadata andbehavioral metrics for a question as parameters to induce deepunderstanding. Specifically, concept_application_tag may be presented asthe metadata and stroke length (stroke_length) and total time(total_time) may be considered as the behavioral metrics.

Referring to FIG. 12, first, a filter may be applied to response-leveldata for all questions to filter questions having a particular questiontag (Q_tag) (e.g., concept_application_tag) at the response level. Next,z-score may be calculated for each behavioral metric at the questionlevel, and the calculated z-score may be normalized to obtain anormalized z-score. The top of FIG. 12 shows Z_Total Time as z-score forthe total_time metric and Z_Norm_Total_Time as a normalized z-score. Thebottom of FIG. 12 shows Z_Stroke_Length as z-score of the stroke_lengthmetric and Z_Norm_Stroke_Length as a normalized z-score. Next, thescores of the metrics may be aggregated at the student level and themetrics may be given a weighted average. For example, in FIG. 12, thetotal time metric may be given a 50% weight and the stroke length metricmay be given a 50% weight to consequently calculate an index of “deepunderstanding” among the CBD factors.

Meanwhile, in regard to questions, a question that takes a long totaltime for an incorrect response and demands a long total input length forthe correct response may be determined to be a question associated withgrit. A question that takes a long total thinking time for the correctresponse and causes many long delays for the correct response may bedetermined to be a question demanding concentration. A question thatgenerates a high number of strokes for the correct response and takes along total time for the correct response may be determined to be aquestion demanding creativity. A question that takes a long initiationtime for the correct response may be determined to be a questionassociated with speed of understanding. A question that demands a longtotal time and is associated with calculation may be determined to be aquestion associated with mental math speed. Additionally, a student'sCBD factor may be evaluated at the test level, and the question's CBDfactor may be evaluated at the question level. For example, it may beevaluated through a test whether the student has grit, and, when athousand people solves various questions, it may be evaluated whichquestions demand grit.

Referring to FIG. 9 again, the diagnostic test apparatus may perform astep (S904) of providing a personalized study plan for the user based ona result of cognitive and behavioral analysis of the user.

For example, the personalized learning module (37) may provide apersonalized study plan for the user by using analysis results of thecognitive and behavioral diagnostic module (36). The personalizedlearning module (37) may identify a particular set of questions thatimproves shortcomings of the user through practice and provide theidentified particular set of questions to the user. Through thealgorithm, the personalized learning module (37) may continuously learnthe user's cognitive and behavioral analysis to make adjustments toimprove quality of recommended practice questions over time.

Additionally, the personalized learning module (37) may perform repeateddiagnosis and analysis of the user through machine learning usingartificial intelligence and, accordingly, provide a personalized studyplan to the user. Thus, it can be said that it is an optimizedpersonalized learning module.

Specifically, the personalized learning module (37) may have an objectof connecting cognitive and behavioral weaknesses identified by thecognitive and behavioral diagnostic module (36). Since the personalizedlearning module (37) may output a study plan at the user's level byusing a method such as a multi-layer collaborative filteringrecommendation engine that uses CBD indices, it can accurately derivepractice questions that improve the user's cognitive weakness areas.Here, the CBD indices may be values for the above-mentioned CBD factors.

The personalized learning module (37) may obtain, as inputs, CBD indicesanalyzed by the cognitive and behavioral diagnostic module (36). Forexample, for each user (student), values of different CBD indices foreach student and each question answered by the student may be providedas inputs. For example, values of CBD indices may be provided as in thematrix format shown below.

Question Student Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 A 4.4 4.4 5.8 6.5 10 4.1 4.11.2 3.1 B 3.5 7.7 7.1 1.4 5.8 6.8 3.0 4.7 1.8 C 4.9 1.8 9.3 2.6 7.3 6.76.6 5.5 4.7 D 7.9 6.8 7.5 2.5 3.9 9.5 9.7 7.4 7.0

Such a matrix may be constructed for each CBD index. Accordingly, forthe eight CBD factors including confidence, grit, reasoning, conceptmemory, deep understanding, calculation ability, ability to understandquestion, and test-taking strategy derived from the cognitive andbehavioral diagnostic module (36), a total of eight matrices may beprovided.

Next, in order to determine i) similarity between questions and ii)similarity between students, the personalized learning module (37) mayapply a similarity function, e.g., cosine similarity, to these matrices.Accordingly, the following two matrices may be created.

A B C D A B C D

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

The similarity values may be calculated by the following expression:

${{similarity}\left( {A,B} \right)} = {{\cos\left( {\overset{\rightarrow}{A},\overset{\rightarrow}{B}} \right)} = \frac{\overset{\rightarrow}{A},\overset{\rightarrow}{B}}{{{{\overset{\rightarrow}{A}}_{2}*\overset{\rightarrow}{B}}}_{2}}}$

For example, as a calculated cosine similarity value gets closer to 1,the similarity between the two vectors gets higher. After calculating i)similarity between questions and ii) similarity between students, thepersonalized learning module (37) may calculate a cognitive gap metricfor a specific question for a specific student by the following formula:

${{Cognitive}{Gap}} = \frac{\sum_{v}\left( {{CBD}\text{?}} \right)}{\sum_{v}\text{?}}$?indicates text missing or illegible when filed

wherein i may be called a question identifier and v may be called astudent identifier.

The cognitive gap metric may be calculated for all of theabove-mentioned eight CBD indices, and an aggregate of eachquestion-student combination may be calculated. These are arranged bysize such that questions having the highest aggregate cognitive gapmetric for respective students may be most recommended for the studentsto improve their weaknesses.

As an another example, the cognitive gap metric may be calculated forall of the above-mentioned eight CBD indices, and the metrics may bearranged by size for each question-student combination such thatquestions having the highest cognitive gap metric for respectivestudents may be most recommended for the students to improve theirweaknesses. For example, when a similarity between questions andsimilarity between student have been calculated, the portion with theleast similarity, i.e., the portion where the CBD factors show thegreatest difference, may be considered first to recommend questions.That is, a question associated with grit (at the question level) may berecommended to a student considered to lack in grit (at the studentlevel), and a question that other students lacking in grit had troublewith may be recommended to the student.

Additionally, by using such a similarity function, a question thatstudents with similar behavioral characteristics usually presented anincorrect answer to and a question that students with a similarperformance (score) presented an incorrect answer to may be recommended.

By using results of analysis of the test diagnostic module (35) and thecognitive and behavioral diagnostic module (36), which are precedingmodules, the personalized learning module (37) may perform machinelearning through repeated analysis and judgment on the user andaccordingly provide a user-customized study plan by recommending themost appropriate questions for the user. For example, through machinelearning, the personalized learning module (37) may identify userslacking in concept memory and/or questions demanding concept memory andalso identify users lacking in deep understanding and/or questiondemanding deep understanding. The personalized learning module (37) mayprovide a user-customized study plan by, for example, providing a set ofquestions identified to be the ones demanding concept memory to the userlacking in concept memory, or a set of questions identified to be theones demanding deep understanding to the user lacking in deepunderstanding. In addition, the personalized learning module (37) may becontinuously learn analysis results from the cognitive and behavioraldiagnostic module (36) so as to be adjusted to improve quality ofrecommended practice questions over time.

The descriptions of the methods and the process flow charts stated aboveare provided as illustrative examples only and are not intended todemand or imply that steps of various embodiments should be performed inthe order in which they are set forth. As a skilled person in the artacknowledges, the steps of the above-mentioned embodiments may beperformed in any order. Expressions such as “accordingly” and “next” arenot intended to limit the order of the steps and are used to guide thereader through the descriptions of the methods.

Various exemplary logic blocks, modules, circuits, and algorithmic stepsdescribed in relation to the embodiments disclosed herein may beimplemented by electron is hardware, computer software, and acombination thereof. For the purpose of clear illustration of theinterchangeability between the hardware and software, various exemplarycomponents, blocks, modules, circuits, and steps are described abovegenerally in terms of their functionality.

Hardware used to implement various exemplary logics, logic blocks,modules, and circuits described in relation to the aspects disclosedherein may be implemented or performed by a general purpose processor, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or anotherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. In addition, in one or more exemplaryaspects, the described functions may be implemented by hardware,software, firmware, or a combination thereof. In case of implementationby software, the functions may be stored in a computer-readable mediumas on one or more instructions or codes or transmitted via acomputer-readable medium, and may be executed by a hardware-basedprocessing unit. Computer-readable media may include computer-readablestorage media corresponding to the type of media such as data storagemedia. In a non-limiting example, such computer-readable storage mediamay include RAMs, ROMs, EEPROMs, CD-ROMs or other optical disk storages,magnetic disk storages, or other magnetic storage devices, flashmemories, or any other media that may be used to store a desired programcode in a form of instructions or data structure or may be accessed by acomputer.

The above-mentioned descriptions of the disclosed embodiments areprovided for any skilled person in the art to carry out and use thepresent invention. Various modifications to these embodiments will beobvious to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the invention. Accordingly, the present invention isnot intended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the following claims, and theprinciples and novel features disclosed herein.

1.-16. (canceled)
 17. A diagnostic test method using a smart pen,comprising: obtaining pen data of the smart pen based on informationinputted by a user for at least one question with the smart pen, theobtaining comprising: identifying a plurality of strokes inputted by theuser on a page with the smart pen, coordinate values of points formingeach of the plurality of strokes, information on time when the pointsare inputted, and writing pressure of the points; detecting delaysassociated with the user inputting the strokes, wherein the delays aredetected based on changes in the coordinate values of the points formingthe plurality of strokes over time; calculating a delay time and anumber of the detected delays; and determining a behavior pattern of theuser based on a number of the plurality of strokes, the delay time, andthe number of the detected delays.
 18. The diagnostic test methodaccording to claim 17, further comprising calculating a total time ofuse of the smart pen by the user.
 19. The diagnostic test methodaccording to claim 18, wherein the calculating the total time comprises:calculating a time of preparation by the user before inputting theinformation on the at least one question; calculating a time when theuser solves the at least one question with the smart pen; andcalculating a total time of input of the information by the user withthe smart pen based on the time of preparation and the time when theuser solves the at least one question.
 20. The diagnostic test methodaccording to claim 19, further comprising: for a first question and asecond question of the at least one question, determining the time ofpreparation by the user by: determining a first time when the smart penwas lifted from the page following a first stroke of solving the firstquestion of the at least one question; determining a second time whenthe smart pen was applied to the page for a second stroke for solvingthe second question of the at least one question, wherein the secondquestion is solved after the first question; and determining the time ofpreparation by subtracting the first time from the second time.
 21. Thediagnostic test method according to claim 17, wherein detecting thedelays further comprise: determining a time difference between a firsttime stamp and a second time stamp, the first time stamp associated witha first stroke of one character and a second stroke of a subsequentcharacter; and determining a pause during a specific time period whenthe user provides the information inputted with the smart pen, the pausedetermined by counting a number of time intervals within the specifictime period in which no stroke is inputted.
 22. The diagnostic testmethod according to claim 17, wherein the determining the behaviorpattern comprises: storing, for a plurality of users associated with theat least one question, analysis information that comprises the number ofstrokes, the delay time, and the number of the detected delays;comparing the stored analysis information with the number of strokes,the delay time, and the number of the detected delays for the user; anddisplaying a diagnosis result for the behavior pattern of the user basedon the comparison.
 23. The diagnostic test method according to claim 22,further comprising: identifying metadata associated with the at leastone question; and determining at least one cognitive and behavioraldiagnostic (CBD) factor from a plurality of CBD factors based on theidentified metadata for the at least one question and the analysisinformation, wherein the at least one CBD factor is selected from atleast one of confidence, grit, reasoning, concept memory, deepunderstanding, calculation ability, ability to understand question, andtest-taking strategy.
 24. A system comprising: one or more computers andone or more storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: obtaining pen data of asmart pen based on information inputted by a user for at least onequestion with the smart pen, the obtaining comprising: identifying aplurality of strokes inputted by the user on a page with the smart pen,coordinate values of points forming each of the plurality of strokes,information on time when the points are inputted, and writing pressureof the points; detecting delays associated with the user inputting thestrokes, wherein the delays are detected based on changes in thecoordinate values of the points forming the plurality of strokes overtime; calculating a delay time and a number of the detected delays; anddetermining a behavior pattern of the user based on a number of theplurality of strokes, the delay time, and the number of the detecteddelays.
 25. The system according to claim 24, further comprisingcalculating a total time of use of the smart pen by the user.
 26. Thesystem according to claim 25, wherein the calculating the total timecomprises: calculating a time of preparation by the user beforeinputting the information on the at least one question; calculating atime when the user solves the at least one question with the smart pen;and calculating a total time of input of the information by the userwith the smart pen based on the time of preparation and the time whenthe user solves the at least one question.
 27. The system according toclaim 26, further comprising: for a first question and a second questionof the at least one question, determining the time of preparation by theuser by: determining a first time when the smart pen was lifted from thepage following a first stroke of solving the first question of the atleast one question; determining a second time when the smart pen wasapplied to the page for a second stroke for solving the second questionof the at least one question, wherein the second question is solvedafter the first question; and determining the time of preparation bysubtracting the first time from the second time.
 28. The systemaccording to claim 24, wherein detecting the delays further comprise:determining a time difference between a first time stamp and a secondtime stamp, the first time stamp associated with a first stroke of onecharacter and a second stroke of a subsequent character; and determininga pause during a specific time period when the user provides theinformation inputted with the smart pen, the pause determined bycounting a number of time intervals within the specific time period inwhich no stroke is inputted.
 29. The system according to claim 24,wherein the determining the behavior pattern comprises: storing, for aplurality of users associated with the at least one question, analysisinformation that comprises the number of strokes, the delay time, andthe number of the detected delays; comparing the stored analysisinformation with the number of strokes, the delay time, and the numberof the detected delays for the user; and displaying a diagnosis resultfor the behavior pattern of the user based on the comparison.
 30. Thesystem according to claim 29, further comprising: identifying metadataassociated with the at least one question; and determining at least onecognitive and behavioral diagnostic (CBD) factor from a plurality of CBDfactors based on the identified metadata for the at least one questionand the analysis information, wherein the at least one CBD factor isselected from at least one of confidence, grit, reasoning, conceptmemory, deep understanding, calculation ability, ability to understandquestion, and test-taking strategy.
 31. A non-transitorycomputer-readable medium storing software comprising instructionsexecutable by one or more computers which, upon such execution, causethe one or more computers to perform operations comprising: obtainingpen data of a smart pen based on information inputted by a user for atleast one question with the smart pen, the obtaining comprising:identifying a plurality of strokes inputted by the user on a page withthe smart pen, coordinate values of points forming each of the pluralityof strokes, information on time when the points are inputted, andwriting pressure of the points; detecting delays associated with theuser inputting the strokes, wherein the delays are detected based onchanges in the coordinate values of the points forming the plurality ofstrokes over time; calculating a delay time and a number of the detecteddelays; and determining a behavior pattern of the user based on a numberof the plurality of strokes, the delay time, and the number of thedetected delays.
 32. The non-transitory computer-readable mediumaccording to claim 31, further comprising calculating a total time ofuse of the smart pen by the user.
 33. The non-transitorycomputer-readable medium according to claim 32, wherein the calculatingthe total time comprises: calculating a time of preparation by the userbefore inputting the information on the at least one question;calculating a time when the user solves the at least one question withthe smart pen; and calculating a total time of input of the informationby the user with the smart pen based on the time of preparation and thetime when the user solves the at least one question.
 34. Thenon-transitory computer-readable medium according to claim 33, furthercomprising: for a first question and a second question of the at leastone question, determining the time of preparation by the user by:determining a first time when the smart pen was lifted from the pagefollowing a first stroke of solving the first question of the at leastone question; determining a second time when the smart pen was appliedto the page for a second stroke for solving the second question of theat least one question, wherein the second question is solved after thefirst question; and determining the time of preparation by subtractingthe first time from the second time.
 35. The non-transitorycomputer-readable medium according to claim 31, wherein detecting thedelays further comprise: determining a time difference between a firsttime stamp and a second time stamp, the first time stamp associated witha first stroke of one character and a second stroke of a subsequentcharacter; and determining a pause during a specific time period whenthe user provides the information inputted with the smart pen, the pausedetermined by counting a number of time intervals within the specifictime period in which no stroke is inputted.
 36. The non-transitorycomputer-readable medium according to claim 31, wherein the determiningthe behavior pattern comprises: storing, for a plurality of usersassociated with the at least one question, analysis information thatcomprises the number of strokes, the delay time, and the number of thedetected delays; comparing the stored analysis information with thenumber of strokes, the delay time, and the number of the detected delaysfor the user; and displaying a diagnosis result for the behavior patternof the user based on the comparison.